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ABSTRACT 

The  Brain  is  key  computational  unit  of  Human  beings  and  it  is  built  of  small  volumetric  units  called  Voxel.  This  is 
a  Survey  paper  that  explores  how  deep  learning  can  be  used  in  Brain  Voxel  classification  from  given  functional  magnetic 
resonance  (fMRI)  image  data. 
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INTRODUCTION 

Artificial  Intelligence  (AI)  visualizes  the  Brain  as  a  deep  Neural  Network.  As  per  brain  imaging  concept,  any 
stimuli  carried  towards  the  cell  bodies  and  these  input  signals  from  dendrites  are  passed  to  neurons  which  produce  an 
output  signal  through  Axons.  Since  this  simulates  deep  neural  network  signal  flow  and  various  analyses  have  proven 
effective  results,  recent  research  trend  is  to  use  Deep  learning  for  Brain  imaging  and  voxel  classifications  from  input  FMRI 
data. 

In  this  aspect  further  sections  summarize  the  basics  of  fMRI,  Brain  voxel  classification  approaches,  deep  learning 
usage  in  Voxel  classification,  tools  used  and  a  case  study  on  deep  learning  architectures  used  in  Brain  Working  Memory 
voxel  classification  domain. 

FMRI,  Voxel,  Population  Receptive  Field  (PRF) 

The  fMRI  (functional  Magnetic  Resonance  Imaging)  uses  BOLD  (Blood  Oxygen  Level  Dependent)  contrast 
technique  to  measure  brain  activity.  The  fMRI  scan  record  information  from  each  brain  voxel  across  time  (Belliveau,  1991) 
in  both  resting  and  active  states.  For  each  stimuli  activity,  the  fMRI  records  BOLD  signal  as  variation  or  contrast  in  Blood 
Oxygen  level  from  its  original  or  baseline  value.  [3] 

A  Voxel  is  a  single  volumetric  unit  in  the  brain.  Approximately  100,000  voxels  are  present  in  the  human  brain’s 
cerebral  cortex. 

Population  receptive  field  (pRF)  is  the  group  of  neurons  responding  stimuli.  The  pRF  mapping  is  processing  of 
mapping  voxel  to  the  cortical  field  as  a  function  of  voxel  parameters  such  as  RF  location  and  size. 
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FMRI  Pre-Processing  Tasks  Prior  to  Voxel  classification 

The  FMRI  data  has  to  be  preprocessed  before  classification.  Skull  scripting  or  Brain  extraction  from  given  3D 
fMRI  medical  image  is  first  important  pre-processing  step.  J.Kleesiek  et  al  have  used  3D  CNN  for  skull  scripting  [2]. 

In  [3]  Fayyaz  Ahmad  and  Iftikhar  used  SPM  tool  that  can  support  fMRI  preprocessing  tasks  such  as  smoothing 
with  Gaussian  kernel,  realignment  for  movement  correction  and  slice  time  correction  activities  and  Contrast  comparison. 
Few  Other  preprocessing  steps  include  Data  augmentation.  Registration,  Bias  Field  correction,  Intensity  Normalization-  Z 
score  computation  in  deep  learning  and  Noise  Reduction. 

Voxel  Encoding  and  Decoding 

In  the  human  brain,  this  process  of  mapping  the  activity  or  response  in  terms  of  stimuli  is  called  Voxel  Encoding. 
In  [1]  Encoding  is  defined  as  the  mapping  n-dimensional  features  related  to  subject’s  perception  to  each  brain  voxel.  Same 
way  Decoding  is  defined  as  mapping  between  n-dimensional  dataset  to  1 -dimensional  target.  The  Voxel  encoding  and 
decoding  together  called  as  Voxel  classification 

POPULAR  VOXEL  CLASSIFICATION  APPROACHES 

MVPA  -  Conventional  Method  of  Voxel  Classification 

The  Multivoxel  Pattern  Analysis  (MVPA)  is  one  of  the  multivariate  decoding  technique  which  classifies  and 
predicts  by  deriving  a  brain  pattern  from  input  data.  Multivariate  regression  approach  is  the  Equivalent  continuous  output 
approach  is  [1  ].  The  pyMVPA  toolbox  supports  below  pattern  classification  tasks 

•  Pattern  discrimination.  [4]-  determine  if  predictor  variable  present  in  given  fMRI  dataset. 

•  Pattern  Localization  [4]  -  By  seed  selection  method  selects  voxels  carrying  sufficient  information  to  drive  the 
classifier.  Use  Searchlight  method  to  identify  and  group  the  correlated  voxels  and  form  a  feature  space  and  ‘brain 
map’. 

•  Pattern  characterization  [4]  -This  will  help  to  learn  the  complex  relationship  between  multiple  input  stimuli, 
voxel  activity,  and  response 

Deep  Learning  Methods  in  Voxel  Classification 

Some  Deep  architectures  used  in  voxel  classification  listed  below 

CNN 

Deep  Learning  based.  Convolution  Neural  networks  (CNN)  architectures  emerged  as  powerful  voxel  encoding 
technique  to  model  human  Visual  Processing,  auditory  and  decision  making  system  [3].  Using  these  encoders  human  brain 
activity  can  be  predicted  by  training  each  hidden  layer  of  CNN  using  low-level  visual  input  called  Pixels  [1]. 

Visualization  Models  -The  quality  of  CNN  feature  extraction,  classification,  and  interpretation  is  proven  accurate  on  MRI, 
fMRI,  X-ray  and  CT  images.  Particularly  for  MRI  and  fMRI  analysis  the  CNN  models  VGG16,  VGG19,  ResNet50, 
Inception,  Inception-ResNet  are  suitable  in  terms  of  various  dataset  inputs,  a  number  of  layers,  loss  optimization,  Training 
time,  Bootstrapping  and  hybrid  mechanisms.  Following  would  be  a  general  approach  of  building  a  Deep  learning  model 
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•  Preprocessing  the  Input  Dataset 

•  ROI  Analysis 

•  Build  and  optimize  a  Deep  model 

•  Classification  and  Prediction 

The  regression  or  classification  Performance  will  be  increased  by  using  suitable  GPU  hardware  and  appropriate 
tuning  of  Deep  architectures  used. 

In  addition,  CNN  visualizations  [5]  help  to  understand  and  design  classifiers.  Max-patch,  activation  map,  Guided 
Backpropagation,  Salient  maps.  Autoencoders,  heat  maps,  and  Deconvolution  are  some  visualization  techniques  that  help 
fine  tuning.  Also,  iterative  training  has  been  proved  successful  in  optimizing  memory  during  training  in  the  unsupervised 
training  of  individual  layers  (Bengio  et  al.,  2007) 

DNN 

Early  studies  on  Voxel  encoding  assumed  brain  representations  and  receptive  fields  as  2D  Gabor  filter.  Later  the 
relationship  between  abstract  feature  space,  activity  space  and  BOLD  signal  established  and  led  to  the  conclusion  that  any 
stimuli  to  Deep  Neural  networks  (DNN)  encoding  model  could  predict  BOLD  response.  [5] 

In  reverse  mode,  the  decoders  estimated  the  input  stimuli  from  the  brain  activity  response.  In  this  method  feature 
vs  voxel,  mapping  is  done  as  per  complexity  gradient.  Ie  voxels  mapped  to  high-level  features  drive  the  final  classification. 
Jack  Gallant  [6]  established  Voxel-Wise  Modelling  and  Decoding  a  voxel  encoding  concept  (VWMD)  as  seen  in  below 
diagram 


Auto  Encoder  -  Nonlinear  Feature  Mapping  Method 

“Deep  learning  architectures  combined  with  unsupervised  feature  learning  backbone  a  low  dimensional  feature 
space  which  then  mapped  to  high  dimensional  input  data”.  [7],  Firat  used  Sparsed  Auto  encoder  for  learning  low 
dimensional  nonlinear  feature  mapping  (NLFM)  to  classify  fMRI  data.  Above  concept  was  used  in  a  Memory  task.  The 
voxels  in  temporal  cortex  memory  were  classified  by  above  technique  as  240  classes. 

Case  Study:  Brain  Working  Memory  Voxel  Classification 

In  the  human  brain,  the  Working  memory  (a  type  of  immediate  memory)  is  responsible  for  Executive  functions 
like  Cognition,  self-regulation,  Planning  and  execution  control.  It  is  a  Higher  Order  thinking  control  which  is  most 
important  in  a  Daily  life.  The  Brain’s  Prefrontal  Cortex,  posterior  parietal  cortex,  and  hippocampal  system  are  key 
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components  of  working  memory.  In  this  case  study,  we  will  highlight  a  few  deep  learning  models  used  in  Working 
Memory  voxel  classification. 

Brodmann  Index  (BA) 

In  Brain,  the  visual  cortex  is  located  in  the  BA17,  BA18,  BA19  and  the  auditory  cortex  is  located  in  BA41  and  42. 
BA  10  is  also  associated  with  Working  memory  and  has  a  causal  relationship  with  BA17  [3].  So  voxel  cluster  covering 
BA10,  BA17,  BA19,  BA  41,  BA42  and  BA47  as  Voxel  cluster  ROI  taken  for  Working  memory  and  decision  making  an 
analysis. 

Types  of  Human  Working  Memory 

•  Visual  Working  memory  -A  cognitive  system  that  temporarily  holds  limited  Visual  information  by  capturing 
what  is  seen  by  the  subject  through  Visual  Cortex  brain  region. 

•  Auditory  Working  memory  -  A  cognitive  system  that  temporarily  holds  limited  auditory  information  heard 
through  auditory  cortex  brain  region. 

Visual  Working  Memory  Voxel  Classification 

In  [1  ]  the  subjects  watch  5  movies  which  is  visual  stimulus  and  hence  the  voxels  in  Visual  working  memory  are 
fired.  The  CNN  is  used  to  extract  features  from  this  video  and  classify  fMRI  brain  voxels  by  transfer  learning  and 
parameter  tuning  the  7th  layer  of  CNN. 

In  [3]  visual  stimuli  given  to  subjects  by  watching  birds  and  an  Artificial  Neural  network  was  used  to  classify  the 
dominant  visual  voxels  in  the  prefrontal  brain. 

In  Visual  Working  Memory  domain  the  LSTM  (Long  Short-Term  Memory)  was  first  introduced,  later  Hubei  and 
Wiesel  introduced  CNN  simulating  human  Visual  cortex.  Given  a  Visual  Video  Stimuli  J.  DOSHI  [8]  CNN  was  used  to 
integrate  separate  experiences  in  working  memory  and  form  episodic  memory.  Similarly,  in  [7]  Auto  encoders  are  used  for 
nonlinear  feature  mapping  during  memory  encoding.  Recently  Hebb-Rosenblatt  memory  architecture  and  Short  Term 
Attentive  Working  Memory  model  (STAWM)  is  introduced  by  combining  CNN  and  Recurrent  Neural  Network  (RNN) 

[9]. 

Auditory  Working  Memory  Voxel  Classification 

In  Auditory  Working  Memory  Domain,  the  Restricted  Boltzmann  Machine  [RBM]  was  used  to  model  Auditory 
Cortex.  CNN  is  used  in  building  voxel  wise  cortical  encoding  model  to  replicate  human  auditory  behavior  and  predict  the 
auditory  cortical  response.  [12],  the  main  difference  here  is  auditory  stimuli.  Compared  to  visual  domain  only  a  few  works 
seen  and  it  is  growing  deep  learning  domain. 

Tools  used  for  FMRI  Voxel  Classification 

Fayyaz  Ahmad  and  Iftikhar  [3]  used  Statistics  Toolbox  in  Matlab  for  fMRI  analysis,  Statistical  parametric 
mapping  (SPM)  package  and  Brain  Tutor.  They  visualized  the  most  dominant  voxel  using  MRIcorn  Software.  Tensor  flow, 
Scikitlearn,  Keras,  python  library  is  also  widely  used  [12] 
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CONCLUSIONS 

By  this  Literature  survey,  we  understood  Deep  learning  is  apt  technology  for  Brain  Analysis  use  cases.  We 
explored  human  Brain’s  structural  and  functional  view  in  terms  of  Brodmann  regions,  fMRI  basics,  fMRI  Preprocessing 
techniques,  and  Brain  Voxel  classification.  Also,  we  have  done  a  detailed  review  of  deep  learning  architecture’s  usage  in 
Brain  functional  encoding\decoding  and  Voxel  classification.  In  specific  the  Working  Memory  brain  function  use  case  is 
reviewed  in  deep  learning  context  in  the  last  section. 
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